1. Technical Field
The present subject matter relates generally to a technology for detecting one or more facial images from an input image, and, more particularly, to a subwindow setting method of detecting one or more face image candidate regions for a face detector for detecting one or more face regions by detecting the face image candidate regions using facial colors and then applying Adaboost for the detected face image candidate regions.
2. Description of the Related Art
Technologies for detecting face regions from images have mainly been aimed at detecting face regions from grayscale images, and have reached a level which can be put to practical use at present. That is, since unique characteristics for defining faces can be sufficiently obtained through grayscale images, facial color information is not inevitably required to be used. However, the inventor of the present invention determined that face regions can be more accurately and quickly detected using a small computational load if facial color information is used when the face regions are detected from images, so that the inventor proposed a method of efficiently using facial color information in a process for detecting face regions.
Korean Patent No. 755800 was applied for by the applicant of the present invention on Jun. 21, 2006 and was registered on Sep. 30, 2007. The invention is entitled “Face Detector using Facial Colors and Adaboost and Detecting Method thereof”. The invention, disclosed in Korean Patent No. 755800, is as follows.
FIG. 1 is a block diagram showing the configuration of a face detector using facial color and Adaboost according to the patent invention disclosed in Korean Patent No. 755800.
The face detector includes an image preprocessor 11, a face detection unit 12, a post-processor 13, a storage unit 14, and a facial color membership function learning unit 15.
The image preprocessor 11 converts the Red-Green-Blue (RGB) system of an input image into a Hue-Saturation-Value (HSV) system, obtains a facial color filter image, from which skin color regions are extracted, using an HS value, obtains a grayscale image from a V value, and then stores the obtained images in the storage unit 14. The image preprocessor 11 respectively integrates the obtained facial color filter image with the grayscale image, obtains an integral facial color filter image and an integral grayscale image, and then stores the obtained images in the storage unit 14. If the integral facial color filter image and the integral grayscale image are used, faster operation of the face detection unit 12 can be attained.
The face detection unit 12 sets a subwindow having an arbitrary size and a location, and then detects the existence of a facial image and a non-facial image for the corresponding subwindow based on a facial color filter image and a grayscale image corresponding to the subwindow. If the corresponding subwindow is determined to be a facial image, information about the location and size of the corresponding subwindow is stored in the storage unit 14. The face detection unit 12 sets a subwindow having a minimum size (for example, 20×20 pixels), detects the existence of a facial image and a non-facial image for the corresponding subwindow while scanning throughout a screen from a upper left portion to a lower right portion. Thereafter, the face detection unit 12 changes the size of the subwindow, and then repeats the detection of the existence of a facial image and a non-facial image for the corresponding subwindow while scanning throughout the screen from the upper left portion to the lower right portion.
The post-processor 13 integrates overlapped locations for results, determined as face images by the face detection unit 12, or performs a process of verifying the results using requirements, and then outputs the verified results to an output unit or stores the verified results in the storage unit 14.
The facial color membership function learning unit 15 learns a facial color membership function for each hue based on a plurality of sample images of faces. The facial color membership function for each hue may be previously learned and then supplied to a user in the state of being stored in the storage unit 14. In this case, the facial color membership function learning unit 15 may be omitted.
FIG. 2 is a flowchart showing the operation of the image preprocessor.
When an input image is input at step S21, the Red, Green, and Blue (RGB) system of the input image is converted into an HSV system at step S22. Further, the facial color membership function M(h(x,y), s(x,y)) for the hue value h(x,y) and saturation value s(x,y) of each pixel is calculated using the facial color membership function for each hue stored in the storage unit, and then a threshold is calculated for the calculation results, thereby obtaining a facial color filter image If(x,y) as shown in the following Equation 1 at step S23.
                                          I            f                    ⁡                      (                          x              ,              y                        )                          =                  {                                                    1                                                                                  if                    ⁢                                                                                  ⁢                                          M                      ⁡                                              (                                                                              h                            ⁡                                                          (                                                              x                                ,                                y                                                            )                                                                                ,                                                      s                            ⁡                                                          (                                                              x                                ,                                y                                                            )                                                                                                      )                                                                              ≥                  θ                                                                                    0                                            otherwise                                                                        (        1        )            
Next, integral operations are performed on the facial color filter image If(x,y) and the grayscale image of the value of color v(x,y) as shown in Equation 2, so that an integral facial color filter image Iif(x,y), an integral grayscale image Lint(x,y), and a grayscale integral square Lint2(x,y) are calculated at step S24.
                                                        I              if                        ⁡                          (                              x                ,                y                            )                                =                                    ∑                                                                    x                    ′                                    ≤                  x                                ,                                                      y                    ′                                    ≤                  y                                                      ⁢                                          I                f                            ⁡                              (                                                      x                    ′                                    ,                                      y                    ′                                                  )                                                    ⁢                                  ⁢                                            L              int                        ⁡                          (                              x                ,                y                            )                                =                                    ∑                                                                    x                    ′                                    ≤                  x                                ,                                                      y                    ′                                    ≤                  y                                                      ⁢                          L              ⁡                              (                                                      x                    ′                                    ,                                      y                    ′                                                  )                                                    ⁢                                  ⁢                                            L              int              2                        ⁡                          (                              x                ,                y                            )                                =                                    ∑                                                                    x                    ′                                    ≤                  x                                ,                                                      y                    ′                                    ≤                  y                                                      ⁢                                          L                ⁡                                  (                                                            x                      ′                                        ,                                          y                      ′                                                        )                                            ·                              L                ⁡                                  (                                                            x                      ′                                        ,                                          y                      ′                                                        )                                                                                        (        2        )            
Further, the calculated integral facial color filter image Iif(x,y), the integral grayscale image Lint(x,y), and the grayscale integral square Lint2(x,y) are stored in the storage unit at step S25.
FIG. 3 is a flowchart showing the operation of the face detection unit.
First, the size and location of a subwindow, which is a target of face detection, are initialized at step S31. In the case in which the coordinate of the upper left point of the subwindow is (xtl,ytl) and the coordinate of the lower right point of the subwindow is (xbr,ybr), the density of the facial color filter image is calculated using an integral facial color filter image Iif(x,y) as shown in the following Equation 3 at step S32.
                    density        =                                                                                                                        I                      if                                        ⁡                                          (                                                                        x                          br                                                ,                                                  y                          br                                                                    )                                                        -                                                            I                      tf                                        ⁡                                          (                                                                        x                          br                                                ,                                                                              y                                                          t                              ⁢                                                                                                                          ⁢                              1                                                                                -                          1                                                                    )                                                        -                                                                                                                                                I                      if                                        ⁡                                          (                                                                                                    x                            tl                                                    -                          1                                                ,                                                  y                          br                                                                    )                                                        +                                                            I                      if                                        ⁡                                          (                                                                                                    x                            tl                                                    -                          1                                                ,                                                                              y                                                          t                              ⁢                                                                                                                          ⁢                              1                                                                                -                          1                                                                    )                                                                                                                              (                                                x                  br                                -                                  x                  tl                                +                1                            )                        ×                          (                                                y                  br                                -                                  y                  tl                                +                1                            )                                                          (        3        )            
If the density calculated at step S32 is higher than a predetermined threshold Φ at step S33, the corresponding subwindow is regarded as a face image candidate, and the process proceeds to the next step such that an integral grayscale image and a grayscale integral square for the subwindow are applied to Adaboost at step S34. Meanwhile, if the density calculated at step S32 is not higher than the predetermined threshold Φ at step S33, it is determined that face detection for the corresponding subwindow is not necessary anymore and then the process proceeds to step S37.
A cascade face detection module, which performed learning using Adaboost, determines whether the subwindow, which is regarded as the face image candidate at step S33, corresponds to a facial image at step S34. In this process, the integral grayscale image Lint(x,y) and the grayscale integral square Lint2(x,y) are used.
The Adaboost face detection module includes a plurality of stages, and each stage is a strong classifier learned using an Adaboost algorithm. Each strong classifier determines whether the corresponding subwindow corresponds to a facial image using an integral grayscale image Lint(x,y) and a grayscale integral square Lint2(x,y). The strong classifiers at the respective stages sequentially verify whether the subwindow corresponds to a facial image and then, if the corresponding subwindow is determined as a face pattern by strong classifiers at all of the stages at step S35, the corresponding subwindow is determined as a face region and the size and location of the subwindow are stored in the storage unit at step S36. However, if the corresponding subwindow is determined not to be a facial image at any one of stages of the Adaboost face detection module at step S35, it is determined that there is no need to perform face verification on the corresponding subwindow, and the process proceeds to step S37.
The Adaboost detection module and the cascade face detection process are described in detail in reference [P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features.” In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Kauai, Hi., 12-14, 2001].
At step S37, if the face verification is performed on subwindows, each having a corresponding size, at all the locations, and if the face verification is performed on all the sizes of subwindows at step S38, the process is terminated.
Meanwhile, if the face verification is not performed on subwindows, each having a corresponding size, at all the locations at step S37, the current location is changed to a subsequent location at step S39, and the process proceeds to step S32. Further, if the face verification is not performed on all the sizes of subwindows, conversion is performed on a size for a subsequent subwindow and the location of the subsequent subwindow is initialized at step S39, and the process proceeds to step S32.
Next, the post-processor 13 integrates overlapped subwindows based on the face detection results detected by the face detection unit, or removes one or more specific subwindows from the detection results based on one or more additional requirements (for example, requirements for the location of a face image within an image and the minimum number of overlapped subwindows) which may be applied depending on application fields. The information about the sizes and locations of the respective detected subwindows is directly provided to a user, or provided to at least one subsequent module, such as a face recognition module, which requires the location and size of a face image afterwards.
As described above, when the face detection unit sets subwindows, the patent invention disclosed in Korean Patent No. 755800 sets subwindows, each having an arbitrary size, and then detects the existence of one or more facial images and non-facial images for the subwindows located at all locations while sequentially setting the subwindows throughout a screen from an upper left portion to a lower right portion. Therefore, since this patent invention sets one or more subwindows even for unnecessary regions that definitely correspond to non-facial images and then detects the existence of one or more facial images and non-facial images for the set subwindows, the conventional system may have a problem in that excessive computational load increases when one or more facial regions are detected from an input image.